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1.0  INTRODUCTION 


i.l  The  Problem 

A  need  exists  for  the  early  consideration  of  logistics  factors  during 
weapon  system  design.  The  reason  is  that  significant  cost  savings  can  be 
realized  if  design  changes  are  made  while  the  design  is  only  on  paper  or  in 
digital  data  banks,  without  need  to  modify  expensive  hardware.  These 
savings  can  be  achieved  during  the  conceptual  phase  of  system  acquisition 
when  the  system  operational  requirements  and  maintenance  concept  are  being 
formulated  and  evaluated. 

If  designers  have  the  means  to  evaluate  the  impacts  of  different  weapon 
system  designs  on  logistics  factors  such  as  manpower,  spares,  training, 
technical  order  documentation,  and  facilities,  they  would  select  the  design 
that  encompasses  the  best  combination  of  the  operational  and  logistics 
factors  at  minimum  cost.  This  is  the  ideal  concept;  achieving  it  in  an 
operational  setting  is  a  different  matter. 

The  Air  Force  Human  Resources  Laboratory  (AFHRL)  has  been  working  the 
problem  of  early  consideration  of  logistics  factors  in  weapon  system  design 
from  a  variety  of  perspectives.  The  Laboratory  pioneered  the  estimation  of 
maintenance  manpower  requirements  for  new  weapon  systems  using  simulation 
models  such  as  the  Logistics  Composite  Model.  AFJ4RL  developed  handbooks  on 
technical  order  acquisition.  It  investigated  ways  to  encourage  engineers  to 
consider  human  resources  in  system  design.  It  developed  system  ownership 
costing  models  for  avionics  systems  and  computer  programs  for  assessing  the 
training  implications  of  weapon  systems.  These  products  have  been  applied 
individually  and  on  a  random  basis  within  the  system  acquisition  process. 
If  individual  applications  could  influence  weapon  system  design,  their 
collective  application  might  influence  design  even  more.  In  1976,  an 
ambitious  project  was  initiated  to  establish  a  mechanism  for  the  coordinated 
application  of  these  human  resources  products.  The  project  became  known  as 
the  Acquisition  of  Supportable  Systems  Evaluation  Technology  (ASSET). 

1.?  ASSET 

.  ^  ASSET  is  a  technology  package  consisting  of  computerized  tools  and 
procedures  which  may  be  used  to  evaluate  the  impacts  of  weapon  system 
designs  on  human  resources,  life  cycle  cost,  training  and  technical  order 
requirements.  ASSET  also  has  the  potential  for  coordinating  the  development 
of  training  programs  and  technical  manuals  during  weapon  system 
development.  -^ASSKT  has  eight  procedures:  (a)  Program  Definition  Analysis, 

(b)  Consolidated  Data  Base  Development,  (c)  Integrated  Task  Analysis, 

(d)  Maintenancei  Action  Network,  (e)  Logistics  Resources  Assessment, 
(f)  Comparability  Analysis,  (g)  Life  Cycle  Cost  Assessment,  and  (h)  Design 
Option  Decision  Tree.  There  are  six  computer  programs:  (a)  Reliability  and 
Maintainability  Model,  (b)  Reliability  and  Maintainability  Cost  Model, 

(c)  Training  Requirements  Analysis  Model,  (d)  Training/Aiding  Matrix, 

(e)  Page  Estimating  Equations,  and  (f)  Personnel  Availability  Model. 
Descriptions  of  each  procedure  and  tool  are  presented  in  Section  3.0. 
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ASSET  was  the  end  product  of  a  four- phased  effort  that  spanned 
approximately  6  years.  The  last  three  phases  were  designed  to  build  on  the 
achievements  of  each  preceding  phase.  Phase  one  was  called  “The  Life  Cycle 
Cost  of  the  C-1301  Weapon  System.”  It  laid  the  foundation  for  constructing 
a  coordinated  human  resources  technology  by  addressing  the  development  of  a 
data  base  for  use  in  establishing  the  support  requirements  of  new  systems. 
It  demonstrated  the  feasibility  of  obtaining  historical  system  data  that 
could  be  used  in  estimating  the  life  cycle  costs  to  be  associated  with  a  new 
but  comparable  system  in  design.  The  case  study  was  the  C-130E  series 
(Hercules)  aircraft.  Five  technical  reports  document  the  C- 130E  study  (see 
References  1-5). 

Phase  two  was  entitled  “The  Integration  and  Application  of  Human 
Resource  Technologies  in  Weapon  System  Design."  It  established  a  basis  for 
integrating  the  technologies  of  system  ownership  costing,  manpower  and 
personnel  estimation,  training  and  technical  order  requirements  into  one 
technology  package.  The  mechanism  for  coordinating  these  separate 
technologies  was  the  consolidated  data  base  concept  developed  in  phase  one. 
Specifications  for  a  consolidated  data  base  were  developed  in  this  phase. 
All  of  the  data  elements  that  ASSET  required  were  to  be  assembled  into  one 
repository.  This  meant  that  the  techniques  for  estimating  system  ownership 
costing  and  the  maintenance  manpower  demands  of  new  weapon  systems  would 
access  data  from  one  location.  Furthermore,  the  data  used  to  generate 
system  ownership  costs  could  be  massaged  for  information  on  the  training  and 
technical  order  requirements  for  that  system. 

During  phase  two,  the  Laboratory  demonstrated  the  coordinated  technology 
package  and  its  consolidated  data  base  during  the  1977-1979  time  frame.  The 
test  case  was  the  Advanced  Medium  Short  Take-off  and  Landing  Transport.  The 
government  engineers  werA  evaluating  various  configurations  of  its  avionics 
and  landing  gear  subsystems  and  they  used  components  of  ASSET  to  aid  them  in 
their  evaluations.  The  transport  was  to  be  the  ASSET  long  term  test, 
evaluation  and  validation  vehicle;  however  the  transport  program  was 
cancelled  in  midstream.  Efforts  to  test  ASSET  were  limited  to  a  modest 
application  in  phase  four.  That  application  is  the  basis  of  this  report. 
Nine  technical  reports  document  the  transport  demonstration  results  and  the 
efforts  to  integrate  the  various  technologies  into  one  package  (see 
References  6-14). 

Phase  three  was  entitled  "Maintenance  Personnel  Availability  Analysis." 
In  phase  three,  the  issue  of  maintenance  personnel  availability  was 
addressed.  The  concern  over  personnel  availability  came  out  of  research  and 
development  (R&D)  findings  in  phase  one.  During  that  phase,  experienced 
C-130E  operations  and  maintenance  personnel  were  interviewed  about  the 
adequacy  of  personnel  to  support  the  aircraft  at  various  AF  bases.  Survey 
results  indicated  that  operational  allocations  were  satisfactory  but  current 
maintenance  personnel  allocations  were  not.  According  to  the  interviews, 
maintenance  personnel  were  stretched  thin  to  keep  squadrons  flying.  There 
appeared  to  be  a  need  for  tools  to  forecast  the  maintenance  personnel 
requirements  of  new  weapon  systems.  Therefore,  phase  three  concentrated  on 
ways  to  accomplish  the  following  objectives:  (a)  estimate  the  future 
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availability  of  maintenance  personnel,  (b)  Identify  possible  difficulties  in 
meeting  future  weapon  system  personnel  requirements,  and  (c)  examine 
corrective  actions  that  could  alleviate  long-run  personnel  shortages  in 
various  job  categories.  The  outcome  was  a  computerized  model  and  an 
application  methodology  for  tackling  the  three  areas  just  described.  The 
model  was  called  Personnel  Availability  Model  (PAM).  Four  technical  reports 
describe  the  PAM  methodology  and  the  R&D  that  went  into  its  development  (see 
References  15-18). 

Phase  four,  entitled  "Test  and  Evaluation  of  Technology  for  Acquiring 
Supportable  Systems,"  was  the  last  phase  and  is  the  subject  of  this  report. 
It  was  labeled  the  test  and  evaluation  sequence  because  it  had  two 
purposes:  (a)  to  determine  ASSET'S  operational  readiness  for  field  use,  and 
(b)  to  evaluate  its  ability  to  consider  human  resources  in  weapon  system 
design.  Although  it  was  intuitively  regarded  that  human  resources  and  other 
logistics  considerations  (spares,  training,  technical  manuals,  etc.)  could 
be  actively  considered  in  system  design  during  the  conceptual  phase,  there 
was  no  consensus  within  the  Laboratory  that  ASSET  was  the  valid  means.  The 
previous  attempt  to  test-run  ASSET  upon  a  system  was  precluded.  The 
transport  program^  mentioned  previously,  was  cancelled  before  ASSET  could  be 
demonstrated  fully  and  its  outputs  verified  against  real  system  data.  The 
next  section  describes  the  experimental  design  for  the  phase  four  study. 
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2.0  TEST  AND  EVALUATION  OESIGN 


2.1  The  Hypothesis 

The  hypothesis  behind  ASSET  was:  that  it  provided  the  logistics 
community,  design  engineers,  logistics  engineers,  and  manpower,  personnel, 
and  training  specialists  with  effective  tools  to  inject  supportability 
considerations  into  weapon  system  design  during  the  conceptual  design 
phase.  This  injection  function  was  realized  through  the  evaluation  and 
selection  of  alternative  design  options.  ASSET  was  assumed  to  have  great 
power  in  evaluating  alternative  design  impacts  on  the  human  resources  and 
other  logistics  considerations. 

The  objective  of  the  test  was  to  determine  if  ASSET  could  be  used  to 
influence  weapon  system  design. 

2.2  The  Test  Bed  and  Application  Technique 

For  the  test  bed,  the  Laboratory  selected  the  APG  66  radar  used  in 
tactical  aircraft  such  as  the  F-16.  The  radar  was  selected  for  the 
following  reasons: 

1.  Expediency  -  The  contractor  selected  to  conduct  the  test  and 
evaluation  had  built  the  radar  and  had  access  to  its  data. 

2.  Relevancy  The  ASSET  package  was  best  suited  for  avionics 
applications  since  some  of  its  components  were  from  avionics  data. 

3.  Research  Opportunity  The  radar  presented  an  alternative  design 
problem. 

The  radar  had  undergone  a  reconfiguration.  The  basic  radar  consisted  of 
three  subsystems  and  six  line  replaceable  units  (LRUs).  The  improved 
version  had  three  subsystems  but  only  five  LRUs.  The  main  difference  was 
the  computer.  It  was  packaged  differently;  instead  of  a  digital  signal 
processor,  it  now  was  a  programmable  one.  Figure  1  displays  both  radar 
configurations . 

ASSET  would  be  used  to  investigate  the  impacts  of  each  alternative  radar 
configuration  on  such  logistics  considerations  as  reliability, 
maintainability,  technical  orders,  and  training  requirements.  Although  this 
application  would  be  purely  academic  since  the  improved  radar  configuration 
had  already  been  selected  and  was  well  into  full  scale  production,  this 
after-the-fact  design  problem  presented  a  unique  opportunity  to  try  out  the 
ASSET  components  in  order  to  evaluate  how  well  or  how  poorly  it  could  assess 
the  logistics  implications  of  system  designs.  The  test  was  to  last  1  month. 

The  same  team  of  contractors  responsible  for  the  conduct  of  the  test  and 
evaluation  contract  effort  would  also  apply  ASSET  to  the  radar 
configurations.  They  were  logistics  engineers  knowledgeable  in  integrated 
logistics  support  modeling  techniques. 


The  application  technique  was  purposely  left  open-ended.  The  logistics 
engineers  were  instructed  to  apply  ASSET  given  an  ASSET  users  manual  and  the 
computerized  techniques  accessible  on  the  computer  facility  at 
Wright-Patterson  AF8.  The  contractors  were  to  approach  the  ASSET  test  and 
evaluation  as  novices,  expecting  the  ASSET  package  and  its  documentation  to 
provide  the  bootstrap  capability  for  implementing  the  package  or  any  of  its 
components.  Data  collection  procedures  were  also  left  vague.  The 
contractors  were  to  collect  the  data  required  to  build  the  ASSET  data  base. 
The  data  collection  effort  was  confined  to  the  contractor's  in-house  sources. 

2.3  The  Evaluation  Criteria  and  Assessment  Scheme 

The  logistics  engineers  were  to  evaluate  each  ASSET  component  against 
four  criteria.  These  criteria  were  designed  to  measure  ASSET'S  operational 
readiness  and  its  efficacy: 

1.  Criterion  No.  1:  Can  the  ASSET  component  contribute  logistical 

information  critical  to  the  selection  of  one  design  option  over 
another? 

2.  Criterion  No.  2:  Is  the  ASSET  component  well  defined?  If  the 

component  is  a  procedure,  does  it  contain  specific  instructions  to 
conduct  the  analysis?  If  a  component  is  a  computerized  technique, 
are  the  data  elements  sufficiently  well-defined  to  preclude  any 
confusion  over  their  meanings? 

3.  Criterion  No.  3:  Are  the  theoretical  assumptions  explicit  and  do 

they  make  intuitive  sense  to  a  logistics  engineer? 

A.  Criterion  No.  A:  Is  the  component’s  documentation  complete  and 

clear  such  that  outside  application  advice  can  be  kept  to  a  minimum? 

No  rating  scheme  was  developed  to  rate  ASSET'S  components.  The 
logistics  engineers  would  subjectively  assess  ASSET  by  each  criterion. 
Their  remarks  would  indicate  if  the  component  was  adequate  or  deficient  in 
meeting  each  criterion.  These  narrative  assessments  would  provide  more 
insight  into  each  evaluation  criterion  than  would  a  numerical  rating  scheme. 

The  thrust  of  this  effort  was  to  test  and  evaluate  ASSET  and  not  two 
radar  design  configurations.  Test  results  for  both  radar  configurations  are 
used  for  illustrative  purposes  only  in  this  report  and  carry  no  significance 
outside  of  this  specific  demonstration. 

Before  delving  into  the  test  application,  it  might  prove  beneficial  for 
the  reader  to  become  familiar  with  the  hypothesized  application  of  ASSET  to 
a  weapon  system  design.  Section  3.0  presents  a  typical  ASSET  application. 
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Baseline  Configuration 


Figure  1.  Equipment  hierarchies  for  radar  designs. 
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3.0  TYPICAL  ASSET  APPLICATION 


Figure  2  displays  at  a  glance  how  an  ASSET  application  is 
conceptualized.  An  ASSET  application  starts  with  the  Program  Definition 
Analysis.  Through  this  process,  the  scope  and  goals  of  the  particular 
application  are  set.  Decisions  as  to  which  procedures  and  models  to  use  are 
also  made.  For  example,  if  the  purpose  was  to  evaluate  the  life  cycle  cost 
implications  of  a  design,  the  Life  Cycle  Cost  Assessment  procedure  and  its 
associated  technique,  the  Reliability  and  Maintainability  Cost  Model,  would 
be  used.  The  ASSET  user  must  learn  as  much  about  the  proposed  weapon  system 
as  possible  including  its  system  and  mission  requirements,  maintenance 
concept,  and  physical  characteristics. 

Once  the  scope  and  goals  are  determined,  the  consolidated  data  base  is 
developed.  Acquisition  of  the  right  type  of  input  data  in  a  timely  manner 
and  the  presentation  of  these  data  in  a  manageable  format  are  important 
parts  of  the  data  base  concept.  The  data  base  stores  all  the  data  that  the 
models  and  procedures  need.  (Figure  3  depicts  the  data  categories.)  Data 
are  stored  in  both  hard-copy  form  and  software  files.  The  data  base  will 
also  store  the  data  generated  by  the  various  models.  The  user  must  use 
ingenuity  in  locating  and  obtaining  data  to  build  a  consolidated  data  base. 
Portions  of  the  data  base  may  be  established  immediately  if  values  for  data 
elements  are  available;  some  data  may  be  derived  from  other  ASSET 
procedures,  such  as  the  Integrated  Task.  Analysis  and  the  Maintenance  Action 
Network. 

ASSET  requires  task  analysis  data  specific  to  the  proposed  design.  This 
task  analysis  is  developed  using  the  Integrated  Task  Analysis  procedure  in 
conjunction  with  the  Maintenance  Action  Network  procedure.  The  task 
analysis  is  driven  by  data  requirements  for  the  maintenance  action  network. 

For  example,  the  user  must  provide  task  data  for  up  to  seven  types  of 
flightline  maintenance  events  and  three  types  of  shop- related  maintenance 
events  for  each  piece  of  equipment.  The  user  must  also  identify  the 
maintenance  personnel  by  Air  Force  Specialty  Code  (AFSC)  and  skill  level  and 
the  must  identify  support  equipment  required  to  perform  the  maintenance 
events.  These  maintenance  events  are  pegged  to  the  generic  Maintenance 
Action  Network  which  serves  as  the  ASSET  operating  foundation. 

An  explanation  of  the  data  requirements  of  the  Maintenance  Action 
Network  should  help  clarify  the  limited  scope  of  the  Integrated  Task 
Analysis.  The  network  represents  a  generalized  maintenance  concept  for  a 
weapon  system  consisting  of  flightline,  and  shop  repair  levels.  The  purpose 
of  the  network  is  to  generate  the  reliability  and  maintainability  parameter 
values  to  be  associated  with  the  new  system  design.  The  reliability  and 
maintainability  factors  critically  influence  other  logistical 
considerations.  In  addition,  the  network  also  aids  the  task  analysis  data 
collection.  ASSET  uses  a  tree  network  as  depicted  in  Figure  A.  The  network 
is  composed  of  seven  generic  flightline  events.  Depot  events  are  not 
considered  explicitly.  This  maintenance  action  network  always  consists  of 
seven  flightlinc  and  three  shop  events.  This  network  is  considered  adequate 
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Figure  3.  Consolidated  Data  Base  (CDB)  components. 
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for  representing  a  basic  maintenance  concept  for  a  weapon  system.  The 
network  may  be  kept  as  a  hard-copy  file  in  addition  to  the  computer-based 
data  files. 

Once  the  maintenance  action  network  is  established,  the  user  moves  into 
the  Logistics  Resources  Assessment  Procedure.  Through  this  procedure,  the 
logistics  and  human  resources  required  by  the  weapon  system  are  identified 
and  investigated.  These  resources  include  such  items  as  support  equipment, 
technical  manuals,  spares,  maintenance  personnel,  facilities  and  training. 
The  burden  of  investigation  rests  with  the  user,  as  this  procedure  requires 
that  the  user  investigate  and  determine  the  supportablity  resource  needs 
under  the  proposed  support  concept.  During  the  Logistics  Resources 
Assessment  Procedure,  one  may  invoke  several  or  all  six  of  the  techniques  to 
help  assess  the  logistics  support  of  the  system  design.  For  example,  the 
page-estimating  relationships  PAGES  can  be  used  to  help  estimate  the  content 
and  quantity  of  technical  order  documentation  to  support  the  system. 

The  next  step  is  the  Comparability  Analysis  procedure.  The 
comparability  analysis  involves  identifying  similar  equipment  to  use  as  a 
basis  for  forecasting  the  resource  requirements  of  the  new  system.  For 
example,  the  failure  rate  data  associated  with  a  similar  baseline  system  can 
be  used  if  those  values  are  adjusted  to  reflect  the  different 
characteristics  of  the  new  system.  Similarly,  the  user  may  invoke  the 
Comparability  Analysis  procedure  to  obtain  a  variety  of  data  for  the  new 
system,  such  as  maintainability  parameter  values  and  system  ownership  cost 
values  to  support  the  ASSET  evaluations. 

After  the  Comparability  Analysis  procedure,  the  user  would  access  the 
Life  Cycle  Cost  Assessment  procedure.  Through  this  procedure,  the  life 
cycle  cost  impact  of  alternative  system  designs  may  be  evaluated.  This 
procedure  uses  the  Reliability  and  Maintainability  Cost  Model  the  ASSET  life 
cycle  cost  model,  to  evaluate  the  economic  costs  of  the  system  design  over 
its  expected  life  time.  This  procedure  promotes  understanding  of  the 
relationships  among  system  performance,  reliability,  maintainability,  life 
expectancy  and  cost  of  complex  systems. 

The  final  procedure  is  the  Design  Option  Decision  Tree.  The  decision 
tree  illustrates  the  various  alternative  design  decisions  as  depicted  in  the 
sample  in  Figure  S.  One  can  annotate  the  decision  tree  with  life  cycle 
costs,  reliability  and  maintainability  values,  and  all  other  data  derived 
from  the  analyses.  The  decision  tree  allows  one  to  view  at  a  glance  which 
alternative  may  pose  the  best  solution  to  the  engineer's  design  problem. 

The  set  of  steps  just  described  above  depict  the  concept  application  of 
ASSET.  The  next  section  describes  the  operational  application  of  ASSET  to 
two  radar  hardware  configurations  and  the  results  obtained. 
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4.0  RESULTS 


Results  of  the  test  and  evaluation  sequence  are  organized  in  the 
following  manner.  First,  a  synopsis  is  presented  of  the  radar  test 
application.  This  is  followed  by  the  criteria  assessments  of  each  ASSET 
component.  The  first  component  tested  and  evaluated  was  the  Program 
Definition  Analysis. 

4.1  The  Procedures 

Program  Definition  Analysis 

Radar  Test  Synopsis:  In  order  to  place  the  radar  system  into  a  context 
for  ASSET  application,  the  logistics  engineers  had  to  invoke  the  Program 
Definition  Analysis  procedure.  In  this  procedure,  the  engineers  had  to 
define  their  radar  system,  and  they  had  to  determine  what  their  analyses 
objectives  were.  To  define  their  radar  system,  they  had  to  develop  a 

general  physical  description  of  the  system,  i.e.,  how  many  subsystems,  line 
replaceable  units,  and  shop  replaceable  units  it  had;  its  operational 
requirements;  its  maintenance  concept  definition;  its  estimated  reliability 
and  maintainability  parameter  values;  its  test  equipment,  maintenance 
manpower,  and  skill  level -requ i rement s .  The  engineers  obtained  data  from 
in-house  sources. 

The  engineers  also  determined  their  analysis  goals.  They  were 
interested  in  evaluating  the  reliability  and  maintainability  of  the  two 
radar  configurations,  their  life  cycle  costs,  and  the  training  and  technical 
order  implications  of  both  configurations.  They  selected  the  following 
procedures  and  techniques  to  assist  them  in  their  analyses: 

(a)  Consolidated  Data  Base  Development,  (b)  Integrated  Task  Analysis, 
(c)  Maintenance  Action  Network,  (d)  Logistics  Resources  Assessment, 
(e)  Comparability  Analysis,  (f)  Life  Cycle  Cost  Assessment,  and  (g)  Design 
Option  Decision  Tree  procedures.  ,  They  selected  the  Reliability  and 
Maintainability  Model,  the  Reliability  and  Maintainability  Cost  Model, 
Page-Estimating  Equations  and  Training/Aiding  Matrix  for  the  techniques. 
The  Personnel  Availability  Model  and  the  Training  Requirements  Analysis 
Model  were  not  selected  because  of  known  technical  difficulties.  (These  are 
discussed  later  in  the  report.) 

The  Assessment: 

The  engineers  could  not  evaluate  the  Program  Definition  Analysis  because 
it  is  more  a  process  than  a  procedure.  A  procedure  should  contain  a 
sequence  of  clearly  delineated  steps  for  doing  an  analysis  in  order  to 
obtain  a  result.  This  program  definition  procedure  does  not  contain  a 
sequence  of  steps,  but  merely  provides  guidelines  for  encouraging  the  R&D 
for  a  weapon  system  design.  How  individual  engineers  conduct  this  R&D  and 
what  areas  they  investigate  are  left  to  their  own  devices.  However,  the 
logistics  engineers  thought  the  intent  of  the  program  definition  analysis 
was  important;  i.e.,  the  logistics  engineer  should  know  as  much  as  possible 
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about  the  weapon  system  program  and  its  potential  impacts  on  a  myriad  of 
logistics  considerations.  Consult  Reference  11  for  additional  information 
on  the  procedure . 

Consolidated  Data  Base  Development 

Radar  Test  Synopsis:  The  logistics  engineers  built  their  consolidated 
data  base  to  support  the  evaluation  of  the  radar  system.  They  required  a 
variety  of  data  such  as  reliability  and  maintainability  data,  system 
ownership  cost  data,  maintenance  event  task  times,  and  equipment  complexity 
data  (such  as  the  number  of  subsystems,  line  replaceable  units  and  shop 
replaceable  units)  for  the  radar's  baseline  and  improved  configuration. 
They  obtained  their  data  from  a  variety  of  engineers  in-house  and  in  a 
variety  of  formats;  some  data  existed  in  hard  copy  form;  some  data  existed 
as  engineering  best- guesses . 

The  Assessment: 

The  consolidated  data  base  was  evaluated  directly  against 
Criterion  No.  2,  (How  well  is  it  defined?)  and  indirectly  against  the  other 
criteria  through  the  ASSET  model  evaluations.  It  was  reasoned  that  the 
model  assessments  would  reflect  the  sufficiency  of  the  consolidated  data 
base. 

Criterion  No.  2  Are  the  data  base  data  elements  well  defined  so  as  to 

preclude  any  confusion  over  their  meanings? 

To  evaluate  Criterion  No.  2,  the  data  base  data  elements  were  evaluated 
in  two  ways.  First,  they  were  evaluated  against  the  Military  Standard 
(MIL-STD)  1388,  which  governs  logistics  support  analysis  (LSA)  and  its 
Logistic  Support  Analysis  Records  (LSAR)  that  document  the  LSA  data. 
Second,  the  data  elements  were  reviewed  for  their  sufficiency  to  analyze  a 
system  design  from  a  variety  of  perspectives. 

The  logistics  engineers  noted  that  the  CDB  and  the  LSAR  share  some 
similarity.  Approximately  30  percent  of  data  elements  are  also  specified  in 
the  LSAR.  The  majority  of  the  data  element  similarities  appear  on  the 
following  LSAR  sheets:  Data  Sheet  B,  Item  Reliability  and  Maintainability 
Characteristics;  Data  Sheet  B2,  Criticality  and  Maintainability  Analysis; 
Data  Sheet  D,  Operation  and  Maintenance  Tasks  Analysis,  and  Data  Sheet  Dl , 
Personnel  and  Support  Requirements.  The  major  difficulty  noted  was  that  the 
data  base  data  element  formats  are  not  compatible  with  LSAR  formats.  LSAR 
data  sheets  cannot  be  directly  loaded  into  the  consolidated  data  base  data 
element  files  without  some  manipulation. 

Regarding  data  elements’  sufficiency,  the  data  base  data  requirements 
seemed  to  cover  a  wide  range  of  logistics  considerations.  (See  Figure  3  for 
the  domain  of  subject  categories.)  The  major  concern  was  the  potential 
nonavailability  of  the  logistics  data  during  the  conceptual  phase  of  system 
acquisition.  It  was  assumed  that  a  user  would  input  best  estimate  data 
until  hard  data  became  available.  This  soft  data  would  constitute  adjusted 
values  obtained  from  comparable  systems  on  which  credible  data  exist. 
(Reference  8  contains  a  detailed  description  of  the  CDB  data  specification.) 
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Integrated  Task  Analysis 


Radar  Test  Synopsis:  The  logistics  engineers  did  not  conduct  a  task 
analysis  for  the  radar  configurations  because  of  time  limitations.  They 
obtained  task  analysis  data  from  training  specialists  and  maintenance 
specialists  within  their  organization.  The  subject  analysis  would  have 
taken  at  least  approximately  2  weeks  for  a  demonstration  designed  for  only  a 
month.  However,  they  reviewed  the  Integrated  Task  Analysis  procedure  and 
prepared  an  assessment. 

The  Assessment: 

Criterion  No.  1  -  Can  the  Integrated  Task  Analysis  contribute  logistical 
information  critical  to  the  selection  of  one  design  option  over  another? 

The  logistics  engineers  gave  high  marks  to  the  procedure.  They  thought 
that  the  Integrated  Task  Analysis  could  provide  information  beneficial  to  a 
decision  maker  if  the  user  has  access  to  comparable  system  data.  The  ASSET 
task  analysis  deviates  from  traditional  maintenance  task  analysis  in  that  it 
is  oriented  more  toward  defining  the  requirements  of  generic  maintenance 
actions  than  toward  the  analysis  of  individual  tasks.  The  word  "Integrated" 
in  the  title  denotes  the  coordinated  development  of  training  courses  and 
technical  orders.  Traditionally,  task  analysis  and  the  determination  of 
technical  order  and  training  requirements  had  been  considered 
independently.  The  ASSET  Integrated  Task  Analysis  provides  mechanisms  for 
their  mutual  development.  Those  mechanisms  include  the  computerized 
technique  called  the  Training/Aiding  Matrix  (to  be  discussed  later).  In 
summary,  the  Integrated  Task  Analysis  procedure  provides  information  that 
may  be  valuable  in  alternative  design  selection. 

Criterion  No.  2  -  Is  the  Integrated  Task  Analysis  well  defined?  Does  it 
contain  specific  instructions  to  conduct  the  analysis? 

Existing  technical  reports  (e.g.,  AFHRL-TR- 80-52,  Feb  1981)  contain 
sufficient  documentation  to  conduct  the  Integrated  Task  Analysis. 

Criterion  No.  3  -  Are  the  theoretical  assumptions  explicit  and  do  they 
make  intuitive  sense  to  a  logistics  engineer? 

The  major  assumption  was  that  one  should  coordinate  the  development  of 
technical  orders  and  training  to  support  a  weapon  system  in  concert  with  the 
identification  and  analysis  of  maintenance  tasks  to  be  associated  with  a  new 
weapon  system.  This  assumption  received  high  marks  from  the  logistics 
engineers . 

Criterion  No.  4  -  Is  the  Integrated  Task  Analysis's  documentation 

complete  and  clear? 

The  existing  users  manual  contains  information  on  the  Integrated  Task 
Analysis;  however,  a  user  would  have  to  consult  References  8  and  13  to  get  a 
full  appreciation  for  the  procedure. 
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Maintenance  Action  Network 


Radar  Test  Synopsis:  The  logistics  engineers  found  the  Maintenance 
Action  Network  useful  in  collecting  the  data.  It  was  especially  helpful  in 
the  annotation  of  the  probability  relationships  among  the  maintenance 
events.  Figure  6  illustrates  the  network  developed  for  each  baseline  radar 
subsystem.  The  decimal  values  at  each  node  are  the  probabilities  of  each 
maintenance  event  occurring  for  each  subsystem  of  the  radar.  In  addition, 
each  node  may  be  annotated  with  the  AFSC  and  skill  level  required  to  perform 
the  tasks  within  each  maintenance  event  and  the  required  support  equipment. 
In-depth  descriptions  of  the  generalized  maintenance  action  network  may  be 
obtained  from  Reference  24. 

The  Assessment: 

Criterion  No.  1  -  Can  the  Maintenance  Action  Network  provide  information 

critical  to  the  selection  of  one  design  option  over  another? 

The  logistics  engineers  gave  the  network  a  marginal  assessment.  It  was 
considered  more  a  data  collection  technique  than  a  decision  aid.  The 
network  serves  as  the  operational  foundation  for  several  ASSET  models,  most 
notably  the  Reliability  and  Maintainability  Model  which  calculates 
reliability  and  maintainability  figures  of  merit  based  upon  this  generic 
network.  The  cost  model  generates  its  life  cycle  cost  assessments  based  on 
this  generalized  network.  The  Training/Aiding  Matrix  derives  its  task  data 
from  it  as  well. 

Criterion  No.  2  -  Is  -the  Maintenance  Action  Network  well  defined?  Is 

enough  information  presented  to  construct  it? 

The  data  elements  are  specific.  There  is  sufficient  information  to 
construct  a  network  and  results  can  be  conveniently  formatted  for  a  user  to 
review.  The  ASSET  network  does  not  encourage  evaluations  of  alternative 
maintenance  support  concepts  which  may  consist  of  maintenance  events  other 
than  the  seven  flightline  and  three  shoprelated  events  already  in  the 
network.  The  network  can  be  modified  by  zeroing-out  the  appropriate  data 
element  fields,  but  it  cannot  be  expanded  to  accommodate  more  network 
branches . 

Criterion  No.  3  -  Do  the  theoretical  assumptions  behind  the  maintenance 

action  network  make  sense  to  a  logistics  engineer? 

The  logistics  engineers  thought  that  the  basic  assumption  made  intuitive 
sense,  up  to  a  point.  A  generalized  network  is  suitable  for  a  weapon  system 
during  the  conceptual  phase  of  system  acquisition,  but  as  a  system  design 
matures,  the  logistics  engineers  concluded  that  the  characteristics  of  its 
support  concepts  may  not  be  captured  in  the  ASSET  network.  In  this  event, 
the  ASSET  evaluations  would  be  rendered  useless. 

Criterion  No.  4  -  Is  the  documentation  complete  and  clear? 

The  documentation  in  the  users  manual  is  sufficient  to  develop  the 
generic  maintenance  action  network. 
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Figure  8.  Baseline  radar  configuration  maintonanca  action  networks. 


Logistics  Resources  Assessment 

Radar  Test  Synopsis:  This  procedure,  which  is  really  more  a  process, 
encourages  the  user  to  apply  the  ASSET  models  to  identify  the  logistics  and 
human  resources  required  to  support  a  weapon  system  configuration.  The 
logistics  engineers  ran  the  Reliability  and  Maintainability  Model  to 
calculate  the  average  maintenance  manhours,  the  average  maintenance  times 
and  the  support  equipment  requirements  of  the  radar  configurations.  They 
ran  the  Reliability  and  Maintainability  Cost  Model  to  obtain  life  cycle  cost 
values  for  both  versions  of  the  radar  system.  They  ran  the  Training/Aiding 
Matrix  and  the  Page  Estimating  Equations  to  assess  the  training  and 
technical  manual  implications  of  the  radar  configurations.  The  results  of 
these  models  provided  the  engineers  with  assessments  of  the  logistics 
requirements  for  the  radar  configurations.  The  results  are  illustrated  in 
Figure  8  at  the  end  of  this  section. 

The  Assessment: 


Criterion  No.  1  -  Can  the  Logistics  Resources  Assessment  provide 
logistical  Information  critical  to  the  selection  of  one  design  option 
over  another  during  the  conceptual  design  phase? 

The  procedure  forces  the  user  to  formulate  the  logistical  support 
considerations  of  the  system  design.  The  procedure  calls  upon  specific 
resource  assessment  techniques  to  establish  a  logistics  resources  profile 
for  the  design.  This  profile  includes  such  considerations  as  manhours, 
skills,  tools,  support  equipment,  spares,  training  and  technical  orders,  and 
system  ownership  cost.  In  this  sense,  the  procedure  can  provide  useful 
logistical  information. 

Criterion  No.  2  -  Is  the  Logistics  Resources  Assessment  well  defined? 
Does  it  contain  specific  instructions  to  conduct  a  resources  assessment 
of  the  system  design? 

The  procedure  is  open-ended  and  relies  on  the  user  to  investigate  the 
logistics  implications  of  the  design  concept.  The  user  may  invoke  the  ASSET 
tools  to  aid  in  the  logistics  resources  assessment  analyses.  The  procedure 
does  not  contain  specific  instructions  for  conducting  a  resource  assessment. 

Criterion  No.  3  -  Are  the  assumptions  behind  the  Logistics  Resources 
Assessment  procedure  explicit  and  do  they  make  intuitive  sense  to  a 
logistics  engineer? 

The  theoretical  statements  are  explicity  stated.  The  theoretical 
objective  is  to  develop  weapon  systems  that  can  be  supported  easily  and 
economically  in  the  field.  To  obtain  this  goal,  the  weapon  system  design 
must  accommodate  as  many  logistics  considerations  as  possible.  The  concept 
of  the  Logistics  Resources  Assessment  procedure  is  to  inject  these 
considerations  into  the  earliest  stage  of  Bystem  development.  The  logistics 
engineers  considered  the  concept  to  be  highly  commendable. 
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Criterion  No.  4  -  Is  the  Logistics  Resources  Assessment  procedure 

documentation  complete  and  clear? 


Documentation  on  this  procedure  is  nonexistent.  There  are  no  guidelines 
to  apply  a  resource  assessment  of  a  system  design  other  than  recommendations 
for  applying  the  various  tools  such  as  the  Reliability  and  Maintainability 
(RM)  Model,  Reliability  and  Maintainability  Cost  Model,  Training/Aiding 
Matrix,  and  Page  Estimating  Equations. 

Comparability  Analysis  Procedure 

Radar  Teat  Synopsis;  The  logistics  engineers  simulated  the  conduct  of  a 
comparability  analysis.  The  reason  was  that  they  already  had  an  improved 
hardware  radar  configuration.  In  essence,  the  improved  radar  configuration 
and  the  data  to  be  associated  with  it  were  derived  from  the  baseline  radar 
system.  The  comparability  analysis  would  address  the  preparation  of  the  new 
system  data.  The  logistics  engineers  outlined  the  steps  in  their  simulated 

comparability  analysis.  Specifically,  the  analysis  would  be  initiated  using 

the  baseline  radar  hardware  configuration  characteristics  as  a  starting 
point.  The  baseline  radar  maintenance  action  network  and  RM  model  results 
would  help  establish  the  basic  reliability  and  maintainability  parameter 
values  for  the  new  system.  The  comparability  analysis  would  also  be 

extended  to  address  the  new  system's  maintenance  tasks  and  other  logistics 

considerations  based  on  the  baseline  radar  maintenance  task  and  logistical 
profile.  The  data  parameter  values  could  then  be  mathematically  adjusted  by 
factors  to  account  for  the  differences  between  the  improved  and  baseline 
radar  configurations.  The  adjustment  factors  could  be  derived  from 
engineering  estimates. 

The  logistics  engineers  evaluated  the  Comparability  Analysis  procedure 
even  though  its  application  was  simulated  for  the  radar. 

The  Assessment: 


Criterion  No.  1  -  Can  the  Comparability  Analysis  procedure  provide 
logistical  information  critical  to  the  selection  of  one  design  option 
over  another  during  the  conceptual  phase  of  system  acquisition? 

Yes,  it  can.  A  comparability  study  is  the  overall  process  used  to 
develop  data  on  newly  designed  systems  by  selecting  operational  equipment 
similar  to  that  of  the  proposed  weapon  system  and  by  adjusting  the  resource 
data  associated  with  baseline  equipment  to  reflect  the  unique 
characteristics  of  the  new  design.  The  analysis  Includes  the  development  of 
maintenance  demand  rates  for  the  new  design  which,  in  turn,  can  be  used  to 
determine  resource  requirements  such  as  manpower,  spares,  and  support 
equipment. 

Criterion  No.  2  -  Is  the  procedure  well  defined?  Does  the  Comparability 
Analysis  procedure  contain  enough  Information  to  conduct  an  analysis? 
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The  ASSET  users  guide  does  not  contain  sufficient  information.  It  is 
assumed  that  the  user  has  the  basic  knowledge  to  do  an  analysis;  the  ASSET 
documentation  merely  provides  recommendations  for  extending  comparability 
analysis  to  other  logistics  factors.  At  best,  the  other  literature  can 
offer  no  more  than  rules-of-thumb  for  doing  an  analysis.  Consult  References 
19,  20,  and  26  for  guidance. 

Criterion  No.  3  Are  the  theoretical  assumptions  explicit  and  do  they 

make  intuitive  sense  to  a  logistics  engineer? 

Yes,  they  do.  There  are  two  major  assumptions.  First,  a  new  system 
design  may  be  developed  based  on  a  comparable  system  in  the  operational 
inventory.  The  majority  of  current  weapon  systems  are  derivatives  of  older 
systems  incorporating  either  new  or  improved  operational  characteristics. 
Second,  comparability  analysis  may  extend  to  other  logistics  considerations 
such  as  manpower,  spares,  training,  and  technical  order  requirements  from 
the  traditional  emphasis  on  maintenance  demand  rates. 

Criterion  No.  4  -  Is  the  Comparability  Analysis  procedure  documentation 

complete  and  clear? 

The  users  guide  cannot  be  the  sole  documentation  source  for  conducting  a 
comparability  analysis.  The  user  must  refer  to  other  sources,  such  as 
indigenous  engineering  personnel  or  References  11  and  19  for  assistance. 

Life  Cycle  Cost  Assessment  Procedure 

Radar  Test  Synopsis:  The  logistics  engineers  applied  the  Life  Cycle 
Cost  procedure  by  invoking  the  Reliability  and  Maintainability  Cost  Model. 
They  applied  this  model  to  both  configurations  of  the  radar  system.  The 
logistics  engineers  did  not  conduct  a  cost  analysis  because  of  insufficient 
guidance  to  conduct  the  analysis.  This  deficiency  is  addressed  below.  The 
cost  results  of  the  Reliability  and  Maintainability  Cost  Model  will  be 
presented  in  the  cost  model  evaluation. 

The  Assessment: 


Criterion  No.  1  Can  the  Life  Cycle  Cost  Procedure  contribute 
logistical  information  critical  to  the  selection  of  one  design  option 
over  another? 

The  procedure  alone  cannot  provide  critical  information.  The  reasons 
are  contained  in  the  following  assessments. 

Criterion  No.  2  -  Is  the  concept  well  defined?  Does  the  Life  Cycle  Cost 
Assessment  procedure  contain  specific  instructions  to  conduct  a  cost 
analysis? 


The  procedure  does  not  contain  sufficient  detail  to  conduct  a  cost 
analysis.  It  is  assumed  that  an  individual  is  aware  of  the  basic  steps 
involved  in  a  cost  analysis,  such  as  (a)  establish  the  objective  or  the 


standard  for  accomplishment,  (b)  identify  all  feasible  alternatives  to  meet 
the  objective,  (c)  formulate  assumptions  about  the  alternatives  being 
evaluated,  (d)  determine  costs  and  benefits,  (e)  compare  costs  and  benefits, 
and  (f)  test  alternatives  under  uncertainty.  The  ASSET  procedure  implies 
these  procedural  steps.  The  logistics  engineers  approached  the  Life  Cycle 
Cost  Assessment  procedure  as  novices  anticipating  guidance  to  prepare  and 
conduct  a  cost  analysis;  however,  they  found  none  in  this  procedure. 

Criterion  No.  3  Are  the  theoretical  assumptions  behind  the  Life  Cycle 
Cost  Assessment  procedure  explicit  and  do  they  make  intuitive  sense  to  a 
logistics  engineer? 

This  criterion  is  best  addressed  during  the  cost  model  evaluation.  The 
Reliability  and  Maintainability  Cost  Model  is  the  principal  tool  in  this 
procedure . 

Criterion  No.  4  Is  the  Life  Cycle  Cost  Assessment  procedure 
documentation  complete  and  clear? 

Documentation  is  not  sufficient  for  a  user  with  no  working  knowledge  of 
cost  analysis  techniques.  The  cost  model  documentation  is  adequate. 

Design  Option  Decision  Tree 

Radar  Test  Synopsis:  The  logistics  engineers  prepared  a  design  option 
decision  tree  to  depict  the  two  design  options  and  the  logistics  resources 
assessment  of  each.  The  data  were  derived  primarily  from  the  ASSET 
application.  The  decision  tree  is  presented  at  the  conclusion  of  this 
section  in  Figure  8  to  summarize  the  findings  of  the  radar  test  application. 

Assessment: 

Criterion  No.  1  Can  the  decision  tree  contribute  logistical 
information  critical  to  the  selection  of  one  design  option  over  another? 

The  decision  tree  can  provide  useful  information.  It  describes  system 
designs  by  graphically  depicting  the  design  options  available  as  the 
designer  progresses  through  a  system  design  problem.  The  decision  tree 
facilitates  early  identification  of  design  options  to  evaluate  appropriate 
options  for  impact  on  human  resources,  logistics  and  cost. 

It  also  provides  an  engineering  paper  trail  for  the  selection  of  one 
design  option  over  another.  For  example,  a  designer  may  subjectively  decide 
to  use  a  design  to  incorporate  a  particular  component  into  an  assembly  with 
little  or  no  apparent  justification.  The  decision  tree  captures  that 
rationale  or  justification  by  the  graphic  depiction  of  the  critical 
charcteristlcs  of  each  alternative  design. 

Criterion  No.  2  -  Is  the  decision  tree  procedure  well  defined?  Does  it 
contain  specific  instructions  to  construct  a  design  option  decision  tree? 
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ASSET  documentation  is  insufficient.  However,  there  are  technical 
reports  available  that  describe  the  mechanics  of  the  decision  tree 
preparation  in  sufficient  detail.  Consult  Reference  21  for  guidance.  The 
ASSET  documentation  treats  the  design  option  decision  tree  at  a  managerial 
level  and  does  not  go  into  the  decision  tree  mechanics. 

Criterion  No.  3  -  Are  the  theoretical  assumptions  behind  the  Design 

Option  Decision  Tree  procedure  explicit  and  do  they  make  intuitive  sense 

to  a  logistics  engineer? 

The  decision  tree  received  high  marks.  The  major  assumption  is:  If  the 
decision  tree  is  deemed  a  feasible  method  for  describing  system  design 
configuration  options,  and  literature  on  other  tests  suggests  that  it  is,  it 

may  provide  a  means  for  the  consideration  of  the  human  resources  and 

logistics  implications  of  system  design.  This  assumption  made  intuitive 
sense  to  the  logistics  engineers. 

Criterion  No.  4  -  Is  the  documentation  complete  and  clear? 

The  ASSET  documentation  is  not  complete.  Decision  tree  definition  and 

intent  are  clear.  Reference  21  should  augment  the  ASSET  users  guide. 

This  completes  the  procedure  evaluations.  The  following  section 
contains  the  model  or  technique  evaluations. 

4.2  The  Models 

Reliability  and  Maintainability  Model 

Programming  Language:  FORTRAN  IV 

Core  Requirement:  142K 

Operation:  Batch 

General  Description:  The  Reliability  and  Maintainability  Model  is  used 
for  estimating  the  average  system  support  resource  requirements  driven  by 
the  reliabilty  and  maintainbility  parameter  values  to  be  associated  with  the 
system  design.  The  Reliability  and  Maintainability  Model  operates  on  the 
generic  Maintenance  Action  Network  concept  to  calculate  the  maintenance 
resource  demands.  The  model  uses  three  figures  of  merit  to  estimate  these 
demands.  These  figures  of  merit  are  (a)  MTTR/KPH  (mean  time  to  repair  per 
1,000  flight  hours),  (b)  MMH/KFH  (direct  maintenance  manhours  per  1,000 
flight  hours),  and  (c)  A}  (system  inherent  availability).  The  figures  of 
merit,  MTTR/KPH  and  MMH/KPH,  are  generated  for  flightline  and  shop 
maintenance.  Detailed  information  regarding  the  Reliability  and 
Maintainability  Model  may  be  obtained  from  Reference  20.  Additional  sources 
for  the  Reliability  and  Maintainability  Model  are  References  22  and  23. 

Radar  Test  Synopsis:  The  logistics  engineers  applied  the  Reliability 
and  Maintainability  Model  to  both  configurations  of  the  radar  system. 

They  obtained  figures  of  merit  for  each  subsystem  of  each  radar 
configuration.  For  example,  figures  of  merit  were  recorded  for  the  radio 
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frequency  subsystems,  the  computer  or  digital  subsystems  and  the  mechanical 
subsystems  for  the  baseline  and  improved  configurations. 

Table  1  shows  the  flightline  figures  of  merit.  The  abbreviations, 
AFRRF,  AFRDI ,  and  AFRMK  represent  the  radio  frequency,  digital,  and 
mechanical  subsystems,  respectively. 

The  Assessment: 

Criterion  No.  1  Can  the  Reliability  and  Maintainability  Model 
contribute  logistical  information  critical  to  the  selection  of  one 
design  option  over  another? 

The  logistics  engineers  thought  that  it  did  contribute  information.  The 
Reliability  and  Maintainability  Model  can  provide  estimates  for  the  average 
maintenance  manhours,  average  repair  times,  and  support  equipment  needs  to 
be  associated  with  a  system  or  its  components.  They  thought  that  the 
figures  of  merit  were  useful  in  identifying  suspected  high  resource 
consumers.  For  example,  in  Table  1  the  radio  frequency  ( AFRRF)  subsystem 
for  both  radar  configurations  had  the  highest  figures  of  merit.  This 
indication  could  spur  the  engineers  to  investigate  the  potential  drivers 
behind  the  figures  of  merit. 

The  Reliability  and  Maintainability  Model  can  also  generate  other 
information  such  as  the  average  repair  time  for  a  component  of  a  system  or 
the  average  maintenance  time  per  task  event  per  subsystem.  The  average 
number  of  manhours  required  to  support  a  particular  maintenance  event  may 
also  be  obtained  from  the  model.  The  Reliability  and  Maintainability  Model 
generates  this  information  in  report  formats.  Detailed  descriptions  of 
these  reports  may  be  obtained  from  Reference  20. 

Criterion  No.  2  Are  the  Reliability  and  Maintainability  Model  data 
elements  well  defined? 

The  data  elements  were  relatively  self-explanatory.  The  engineers 
prepared  a  set  of  53  cards  each  for  the  baseline  and  improved  radar 
configurations.  The  logistics  engineers  experienced  little  or  no  difficulty 
in  providing  input  values  for  approximately  41  data  elements  to  operate  the 
Reliability  and  Maintainability  Model.  A  print-out  of  the  data  inputs  and 
data  values  may  be  obtained  from  Reference  20. 

The  Reliability  and  Maintainability  Model  generates  batch  output  from 
card  input.  Feedback  can  occur  in  as  fast  as  several  minutes  or  as  long  as 
an  hour,  depending  on  the  computer-based  traffic.  The  Reliability  and 
Maintainability  Model  operates  on  a  mainframe  computer  at 
Vright-Patterson  AFB . 

Criterion  No.  3  -  Are  the  theoretical  assumptions  explicit  and  do  they 
make  intuitive  sense  to  a  logistics  engineer? 


Table  1.  Flightline  Level  FOM-Baseline  and  Improved  Configurations  (Identical  Results) 


Subsystem 

1 

1 
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MTTR/KFH 

1 

MMH/KFH  | 

Ai 

AFRRF 
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4.58 

1 

4.58  | 

99.5 

AFROI 

1 

i 

3.40 

3.40  ! 

99.6 

AFRME 

1 

1 

0.78 

0.78  : 

99.9 

AFRRF  radio  frequency 

AFRDI  computer  subsystem 

AFRME  mechanical  subsystem,  i.e.,  antenna, rack. 

MMH/KFH  (Mean  Time  to  Repair  per  1000  flight  hours). 
MTTR/KFH  (Direct  Maintenance  Man  Hours  per  1000  flight  hours). 
Aj  (System  Inherent  Availability). 


The  major  operating  assumption  is  that  one  can  determine  the  maintenance 
requirements  of  a  system  from  an  analysis  of  the  figures  of  merit;  namely, 
Mean  Time  to  Repair  per  1000  flight  hours  and  Maintenance  Manhours  per  1000 
flight  hours.  There  is  serious  concern  that  one  cannot  translate 
maintenance  manhours  directly  into  the  numbers  of  maintenance  personnel 
required  to  support  the  system  once  released  to  the  operational  inventory. 
Also,  maintenance  resources  demand  may  not  be  adequately  estimated  by  an 
average  value  model  such  as  the  Reliability  and  Maintainability  Model  since 
it  is  driven  by  a  generalized  support  concept.  The  generalized  maintenance 
action  network,  may  be  modified  by  inserting  zeroes  for  data  element  fields. 
For  example,  if  a  user  were  interested  only  in  organizational  maintenance, 
the  intermediate  shop  data  fields  would  be  zeroed- out.  The  network  may  not 
be  manipulated  structurally  to  reflect  new  maintenance  events  specific  to  a 
new  weapon  system  or  new  support  concept.  The  logistics  engineers  concluded 
that  the  Reliability  and  Maintainability  Model  may  be  useful  as  a  check 
against  other  analyses  during  conceptual  design  studies  but  should  not  be 
used  as  the  sole  source  for  maintenance  requirements  determination. 

Criterion  No.  4  -  Is  the  Reliability  and  Maintainability  Model 

documentation  complete  and  clear? 

It  was  determined  that  the  documentation  was  complete  and  clear.  The 
users  guide.  Reference  20,  contains  all  the  data  input  and  output 
requirements  for  the  Reliability  and  Maintainability  Model. 

Reliability  and  Maintainability  Cost  Model 

Programming  Language:  FORTRAN  IV  EXTENDED 

Core  Requirement:  125K 

Operation:  Interactive  and  Batch  Output  Reports 

General  Description:  The  Reliability  and  Maintainability  Cost  Model  is 
a  cost  accounting  model  and  a  companion  to  the  Reliability  and 
Maintainability  Model.  It  computes  recurring,  nonrecurring,  and  disposable 
costs  to  be  associated  with  a  new  weapon  system  based  on  the  reliability  and 
maintainability  values  generated  by  the  Reliability  and  Model.  The  cost 
elements  are  depicted  in  Figure  7.  Portions  of  the  Reliability  and 
Maintainability  Cost  Model  can  be  adjusted  to  conduct  sensitivity  analyses. 
The  Reliability  and  Maintainability  Cost  Model  generates  costs  in  constant 
year  dollars.  The  user  may  input  inflation  rates  and  a  discount  rate  to 
account  for  the  time  value  of  money.  Details  on  the  cost  model  and  its 
operation  can  be  obtained  from  Reference  20. 

Radar  Test  Synopsis:  The  Reliability  and  Maintainability  Cost  Model  was 
used  to  study  the  life  cycle  cost  sensitivity  of  the  two  radar 
configurations  to  increases  in  their  individual  reliability  and  unit  costs. 
Mean  flight  hours  between  maintenance  actions  (MFHBMA)  and  unit  cost  were 
increased  20%.  Tables  2  and  3  summarize  the  results.  The  logistics 
engineers  obtained  data  from  in  house  sources.  The  life  cycle  values  were 
discounted  by  10%  to  reflect  the  present  value  of  money.  Table  2  depicts 
the  life  cycle  costs  for  the  baseline  configuration.  The  total  recurring 
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Figure  7.  RMCM  cost  elements. 
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cost  did  not  change  significantly  with  a  20%  increase  in  reliability. 
Nonrecurring  cost  reflected  an  increase,  however,  from  approximately  $33 
million  to  $36  million.  The  total  life  cycle  cost  over  an  Inventory  usage 
period  of  14  years  increased  from  approximately  $70,000  to  $74,000  as  a 

result  in  the  increases  in  reliability  and  unit  cost.  The  same  held  true 
for  the  improved  configuration.  However,  the  improved  radar  design  incurred 
higher  nonrecurring  costs.  Total  life  cycle  cost  was  substantially  higher; 
from  approximately  $159  million  to  $172  million.  Thus,  the  total  life  cycle 
costs  for  both  radar  configurations  were  higher  when  reliability  and  unit 
cost  were  increased  20%.  Table  4  displays  the  initial  life  cycle  costs 

estimates  for  both  radar  configurations. 

The  Assessment: 

Criterion  No.  1  -  Can  the  Reliability  and  Maintainability  Cost  Model 

contribute  logistical  information  critical  to  the  selection  of  one 

design  option  over  another? 

Its  best  contribution  is  its  cost  sensitivity  analysis  feature  that  car 
be  used  to  evaluate  weapon  systems  with  uncertain  design  characteristics. 
The  user  can  mafce  upward  or  downward  adjustments  to  any  number  of  parameter 
values  such  as  MTBF  or  unit  cost  to  analyze  the  anticipated  effects  on 
downstream  costs. 

Criterion  No.  2  -  Are  the  Reliability  and  Maintainability  Cost  Model 

data  elements  well  defined? 

The  data  elements  are  not  fully  defined  since  an  indepth  data  element 
dictionary  does  not  accompany  the  model.  The  data  element  dictionary  that 
exists  gives,  at  a  minimum,  the  name  of  the  data  element  and  its  field 
format,  and  sketchy  data  element  descriptions.  However,  a  user  can  still 
operate  the  model  and  derive  some  benefits  from  it,  especially  from  the 
interactive  sensitivity  analysis  feature. 

The  majority  of  the  Maintainability  Cost  Model  data  elements  are  derived 
from  cost-estimating  relationships.  The  remainder  consist  of  standard 
values  obtained  from  Government  sources  and  historical  estimates  derived 
from  comparable  system  historical  cost  experience. 

Since  the  Reliability  and  Maintainability  Cost  Model  is  an  interactive 
program,  a  decision  maker  can  get  almost  instantaneous  feedback.  The 
Reliability  and  Maintainability  Cost  Model  can  also  operate  on  the 
Reliability  and  Maintainability  Model  data  base  which  precludes  data  base 
construction  from  scratch.  The  only  portion  that  requires  manual  loading  is 
the  cost  data.  The  logistics  engineers  prepared  approximately  40  data  cards 
bearing  cost  values  to  run  the  cost  model  for  one  radar  configuration. 

Additionally,  the  cost  model  provides  a  computer  printout  of  the 
reports.  Included  in  these  reports  are  life  cycle  costs  by  subsystem  and 
LRU  contributions.  Consult  Reference  20  for  further  information. 


Table  2.  Life  Cycle  Cost  -  Baseline  Configuration  for  Radar  with  Three  Subsystems/Six  LRUs 


i - r 

1  Life  Cycle  Cost  Categories  | 

1  1 

Static 

1  1 

1  Perturbed  (Reliability  &  Unit  l 

1  Cost  Increased  20%)  1 

i  1 

Recurring  i 

l  * 

|  Support  l 

1  1 

1  Flight  Line  Maintenance  i 

$  31,063 

1  $  28,237  | 

1  Shop  Maintenance  1 

$  214,069 

|  $  194,596  | 

1  Depot  Maintenance  ■ 

$  236,302 

$  230,497  1 

1  Spares  ; 

$  342,594 

[  $  365,394  1 

*  Other  . 

$30,777,602 

|  $30,775,461  1 

Operation  1 

$  6,024,045 

|  $  6,02.1,045  ' 

!  Subtotal  1 

$37,625,675 

1  $37,618,230  ! 

i  1 

1  i 

1  Non-Recurring  1 

1  . 

•  , 

1  R&D  I 

$  0 

|  0  | 

1  System  Investment  l 

$17,137,162 

1  $20,074,961  j 

1  Support  Investment  1 

$15,222,745 

1  $15,968,143  1 

I  Subtotal  1 

.  i 

$32,359,907 

1  $36,043,104  1 

(  l 

.  Disposal  1 

- L_ 

$  0 

1  0  1 
1  1 

I 


TOTAL 


$09,985,582 


$73,661,334 


Table  3.  Life  Cycle  Cost  -  Improved  Configuration  (Radar  With  Three  Subsystems/Five  LRUs) 


Life  Cycle  Cost  Categories 

1 

1  Static 

I 

1  Perturbed  (Reliability  & 

1  Unit  Cost  Increased  20%) 

i 

Recurring 

1 

1 

1 

| 

Support 

1 

l 

Flight  Line  Maintenance 

1  $  31,062 

1  $  28,237 

Shop  Maintenance 

1  $  266,258 

j  $  242,032 

Depot  Maintenance 

|  $  247,380 

1  $  240,569 

Spares 

$  1,611,104 

,  $  1,731,183 

Other 

,  $  41,089,992 

|  $  41,088,154 

Operation 

1  $  6,024,045 

l  $  6,024,045 

Subtotal 

1 

1  $  49,269,841 

1  $  49.354,220 

Non-Recurring 

1 

1 

1 

R&O 

!  $  0 

1  *  0 

System  Investment 

1  $  56.669,037 

1  $  66.383,729 

Support  Investment 

1  $  53,055,327 

<  $  56,591,531 

Subtotal 

1  $  109,724,364 

I 

,  $  122,975,260 

Disposal 

1  $  0 

|  $  0 

TOTAL 

!  $  158,994,205 

i 

i  $172,329,482 

i 

Table  4.  Life  Cycle  Cost  Comparison 


ICC 

CATEGORIES 


Weapon  System  Radar  Configuration 


Baseline 


Improved 


$31,601,630 
$  6,024,045 


$43,245,796 
$  6,024,045 


Subtotal 


$37,625,675 


$49,269,841 


Non-Recurring 

R&D 

System  Investment 
Support  Investment 


$  0 
$17,137,162 
$15,222,745 


$  0 

$56,669,037 

$53,055,327 


Subtotal 

Disposal 

Total 


$32,359,907 
$  0 


$109,724,364  j 

$  0  I 

_ I 

I 


$69,985,562 


$156,994,205 


j 


Criterion  No.  3  -  Are  the  Reliability  and  Maintainability  Cost  Model 

assumptions  explicit  and  do  they  make  intuitive  sense  to  a  logistics 

engineer? 

The  assumptions  are  stated  explicitly.  The  logistics  engineers 
considered  the  cost  model  assumptions  concerning  the  general  scenario, 
spares  investment,  and  its  cost  elements  plausible. 

For  the  general  scenario,  the  cost  model  considers  a  uniform  level  of 
system  activity  (such  as  flying  hours)  at  each  operating  base.  The  model 
assumes  that  air  bases  are  identical  with  respect  to  environmental  effects 
on  equipment  failure  rates  and  logistics  support. 

With  spares  investment,  the  model  computes  the  ipares  stock  level  and 
pipeline  quantities  to  support  the  peak  level  of  aircraft  activity  and  peak 
base  flying  hours  rather  than  any  incremental  buildups.  It  is  not  known  if 
the  peak  period  donates  wartime  surge.  The  model  assumes  that  inventories 
of  spare  LRUs  are  located  at  each  of  the  bases,  consistent  with  the  demand 
rate  for  replacement  parts  at  the  bases . 

There  are  two  assumptions  regarding  the  model  cost  elements.  First,  the 
model  computes  life  cycle  cost  in  constant  dollars.  Second,  the  model 
specifically  computes  only  those  logistics  support  costs  associated  with  the 
weapon  system,  subsystem  and  LRU  indenture  levels.  The  SRUs  are  derived 
through  implicit  consideration  of  their  relationship  to  the  repair  of  a 
given  LRU.  For  example,  average  costs  of  SRUs  are  computed  based  on  the 
failure  rates  of  the  LRUs. 

Regarding  maintenance  costs,  the  cost  model  calculates  costs  based  on  a 
support  concept  driven  by  equipment  failures  in  the  same  way  that  the 
Reliability  and  Maintainability  Model  computes  maintenance  resource 
demands.  To  compute  the  labor  costs  for  scheduled  maintenance,  the 
maintenance  events  must  be  treated  as  if  they  were  corrective  maintenance 
actions.  The  reliability  value  can  be  interpreted  as  a  function  of  the 
periodicity  of  the  scheduled  action  in  respect  to  the  system  operational 
scenario. 

Criterion  No.  4  -  Is  the  cost  model  documentation  complete  and  clear? 

It  was  determined  that  the  documentation  is  sufficient  to  exercise  the 
model  and  to  analyze  its  results. 

Paae-Estimatina  Equations 

Programming  Language:  FORTRAN  IV 

Core  Requirement:  125K 

Operation:  Interactive 

General  Description:  These  equations  estimate  the  technical  order 
requirements  of  a  new  system.  Specifically,  it  estimates  the  number  of 
pages  and  the  types  of  pages  (narrative  or  illustrations)  for  job  guides. 
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fault  isolation  and  general  system  manuals.  It  can  also  provide  estimates 
for  conventional  technical  manuals  which  are  composed  primarily  of 
narrations  and  theory. 

Data  inputs  are  simple;  the  program  requires  the  number  of  systems, 
subsystems,  LRUs  and  SRUs  in  the  design.  The  estimating  relationships  in 
this  program  are  divided  into  electrical  and  mechanical/hydraulic 
categories.  That  is,  one  set  of  equations  may  be  selected  to  generate 
technical  order  estimates  for  electrical  systems;  another  set  may  be  used 
for  mechanical/hydraulic  technical  order  requirements. 

Radar  Test  Synopsis:  The  logistics  engineers  applied  the  electrical 
system  equations  to  both  configurations  of  the  radar.  Their  input  data 
consisted  of  the  number  of  subsystems,  LRUs,  and  SRUs  contained  in  each 
configuration.  For  the  baseline  radar  there  were  three  subsystems,  six 
LRUs,  and  72  SRUs.  For  the  improved  radar  configuration,  there  were  three 
subsystems,  five  LRUs,  and  58  SRUs. 

Logistics  engineers  entered  these  data  into  the  equation  program  through 
the  interactive  mode.  They  invoiced  the  electrical  system  equations  which 
began  to  process  the  data  and  to  generate  the  estimates. 

The  number  of  conventional  technical  order  pages  to  support  the  baseline 
configuration  were  calculated  first.  The  conventional  technical  orders 
would  be  used  at  the  organization  and  intermediate  maintenance  levels.  The 
logistics  engineers  calculated  45  pages  for  the  organizational  technical 
manual  and  531  pages  for  the  intermediate  maintenance  manual.  For  the 
improved  radar  configuration,  they  estimated  39  pages  and  432  pages  for  the 
organizational  and  intermediate  maintenance  manuals,  respectively.  Table  5 
depicts  the  conventional  manual  page  counts.  Page  content  is  also  indicated 
in  the  table.  Additionally,  page  content  and  page  quantity  is  segregated 
into  troubleshooting  (TS)  and  nontroubleshooting  (NTS)  tasks  for  both  the 
flightline  and  the  shop. 

The  page  estimates  for  the  task-oriented  manuals  were  then  calculated 
for  the  baseline  and  improved  radar  configurations.  The  results  appear  in 
Table  6.  For  the  baseline  radar,  the  logistics  engineers  estimated  165 
pages  for  organizational  manuals  and  531  pages  for  the  intermediate 
maintenance  manual.  For  the  improved  radar  configuration,  they  calculated 
140  pages  and  432  pages,  respectively. 

The  Assessment : 


Criterion  No.  1  Can  these  estimating  equations  provide  logistical 
information  critical  to  the  selection  of  one  design  option  over  another? 

The  logistics  engineers  rated  the  equations  as  marginal.  The  equations 
can  provide  rough  estimates  of  the  technical  order  page  requirements  a 
system  may  generate  using  very  superficial  data  such  as  the  number  of 
components  that  comprise  the  system.  This  is  useful  during  conceptual 
design  studies;  however,  as  the  system  design  matures,  it  may  be  rendered 
grossly  inadequate  to  provide  specific  techical  order  estimates.  The 
following  evaluations  may  help  clarify  this  assessment. 
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Table  5.  PAGES  Conventional  Manual  Estimates 

TECH  MANUAL  CONTENT  ESTIMATE 
ELECTRONICS  -  CONVENTIONAL 


PAGE  TYPE 

NARRATIVE 
HALFTONE  ART 
HALF  TONE  EXPLOSION 
ELECTRONIC  LINE  ART 
explooeo  line  art 

FAULT  ISOWTTON  CHART 

fault  isolation  schematic  block 
access  line  art 

FAULT  ISOLATION  SCHEMATIC  FLOW 
FAULT  ISOLATION  SCHEMATIC  MECH/H 

JOB  6UI0E  ILLUSTRATIONS 
TOTAL 

SUBSYSTEMS  3 

LRU  6. 

GRU  70 


TS 

SHOP  F/L 
12  38  18 

6  42  3 

0  38  0 

6  ISO  0 

0  12  0 

C  0  0 

0  0  0 

COO 

o  0  0 

0  0  o 

o  o  o 

o  0  o 

24  282  21 


Baseline  Configuration 


TECH  MANUAL  CONTENT  ESTIMATE 
ELECTRONICS  -  CONVENTIONAL 


PAGE  TYPE 

NARRATIVE 
HALF  TONE  ART 
HALF  TONE  EXPLOSION 
ELECTRONIC  LINE  ART 
EXPLOOEO  LINE  ART 
FAULT  ISOLATION  CHART 
FAULT  ISOLATION  SCHEMATIC  BLOCK 
ACCESS  LINE  ART 

FAULT  ISOLATION  SCHEMATIC  FLOW 
FAULT  ISOLATION  SCHEMATIC  MECH/  H 
JOB  6UI0E  NARRATIVE 
JOB  GUIDE  ILLUSTRATIONS 

TOTALS 

SUBSYSTEMS  3. 

LRU  5. 

SRU  58. 


SHOP 

32 

34 

32 

121 

10 

0 

0 

0 

0 

0 

0 

0 

229 


NTS 

SHOP 

129 

39 

3 

78 

0 

0 

0 

0 

0 

0 

0 

0 

249 


SHOP 

105 

32 

3 

63 

0 

0 

0 

0 

0 

0 

0 

0 

203 


The  Improved  Configuration 
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labia  6.  PAGES  Task-Oriented  Manual  Estimates 

TECH  MANUAL  CONTENT  ESTIMATE 
ELECTRONICS  -  TASK-ORIENTED 


PAGE  TYPE 

F/L 

TS 

SHOP 

NARRATIVE 

9 

39 

HALF  TONE  ART 

0 

42 

HALF  TONE  EXPLOSION 

0 

39 

ELECTRONIC  LINE  ART 

0 

150 

EXPLODED  LINE  ART 

0 

12 

FAULT  ISOLATION  CHART 

18 

0 

FAULT  ISOLATION  SCHEMATIC  BLOCK 

6 

0 

ACCESS  LINE  ART 

12 

0 

FAULT  ISOLATION  SCHEMATIC  FLOW 

0 

0 

FAULT  ISOLATION  SCHEMATIC  MECH/  H 

0 

0 

JOB  GUIDE  NARRATIVE 

0 

0 

JOB  GUIDE  ILLUSTRATIONS 

0 

0 

TOTAL 

SUBSYSTEMS  3. 

LRU  6. 

SRU  72. 

45 

282 

Baseline  Configuration 

TECH  MANUAL  CONTENT  ESTIMATE 
ELECTRONICS  -  TASK-ORIENTED 
PAGE  TYPE  TS 

F/L  SHOP 


NARRATIVE 

3 

32 

HALF  TONE  ART 

0 

34 

HALF  TONE  EXPLOSION 

0 

32 

ELECTRONIC  LINE  ART 

0 

121 

EXPLODED  LINE  ART 

0 

10 

FAULT  ISOLATION  CHART 

16 

0 

FAULT  ISOLATION  SCHEMATIC  BLOCK 

6 

0 

ACES S  LINE  ART 

10 

0 

FAULT  ISOLATION  SCHEMATIC  FLOW 

5 

0 

FAULT  ISOLATION  SCHEMATIC  MECH/H 

0 

0 

0 

0 

JOB  GUIDE  ILLUSTRATIONS 

0 

0 

TOTALS 

SUBSYSTEMS  3. 

LRU  5. 

SRU  58. 

40 

229 

NTS 


F/L 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

60 

60 


SHOP 

129 

39 

3 

78 

0 

0 

0 

0 

0 

0 

0 

0 


120  249 


NTS 

F/L  SHOP 

0  105 

0  32 

0  3 

0  63 


0 

0 

0 

0 

0 

0 

50 


0 

0 

0 

0 

0 

0 

0 


50  0 


100  203 


Improved  Configuration 
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Criterion  No.  2  -  Are  the  data  elements  well  defined? 

Its  data  elements  are  not  well  defined.  Although  it  requires  only  a 
parts  count  for  either  an  electrical  or  mechnical/hydraulic  system,  it  does 
not  define  the  data  collection  methodology.  Furthermore,  users  must  create 
their  own  definitions  regarding  system,  subsystem,  LRU,  and  SRU  composition. 

The  equations  can  generate  output  within  a  few  minutes  once  the  data  are 
loaded  into  the  program.  Data  can  be  loaded  manually  or  the  Reliability  and 
Maintainability  Model  data  base  may  be  accessed  to  run  these  equations. 

Criterion  No.  3  -  Are  the  assumptions  behind  the  Page  Estimating 

Equations  explicit  and  do  they  make  intuitive  sense  to  a  logistics 

engineer? 

The  theoretical  statement  is  explicit.  The  assumption  states  that 
equipment  complexity  is  proportionally  related  to  the  number  of  technical 
order  pages  required  to  support  a  system.  It  is  also  assumed  that  different 
kinds  of  illustrations  (e.g.,  line  art,  schematics  and  block  diagrams)  are 
directly  correlated  with  equipment  complexity.  These  assumptions  have  not 
been  rigorously  tested  to  merit  any  measure  of  confidence.  Care  should  also 
be  exercised  if  the  Page  Estimating  Equations  are  applied  to  a  variety  of 
aircraft.  Its  estimating  relationships  were  derived  from  tactical  fighter 
maintenance  manual  data  and  are  best  suited  for  tactical  fighter  weapon 
system  applications.  Application  to  systems  other  than  fighter  aircraft  may 
require  reestimation  of  the  parameter  coefficients. 

Criterion  No.  4  -  Is  the  documentation  complete  and  clear? 

The  documentation  is  not  complete  in  terms  of  a  specified  methodology. 
The  documentation  is  not  clear  in  terms  of  clearly  stated  results.  The 
equations  estimate  total  page  quantities.  They  do  not  estimate  page  content 
and  page  quantity  for  a  specific  maintenance  manual  type;  they  force  the 
user  to  aggregate  the  pages  into  the  desired  manual,  either  a  job  guide  or  a 
combined  fault  isolation  and  general  system  manual.  There  is  no  rationale 
to  justify  the  latter  combination.  Reference  20  is  the  documentation  source 
for  these  equations. 

Trainina/Aiding  Matrix 

Programming  Language:  FORTRAN  V 

Core  Requirement:  125K 

Operation:  Interactive 

General  Description:  The  Training/Aiding  Matrix  (TAM)  indicates  the 
relative  emphasis  that  should  be  placed  on  technical  training  and  the 
technical  manual  for  proper  performance  of  maintenance  tasks  assigned  to  a 
new  weapon  system.  The  matrix  uses  ratios  of  numbers  consisting  of  l's, 
2's,  and  3’s  to  indicate  whether  technical  training  or  technical  manuals 
should  be  given  light,  medium,  or  heavy  emphasis  to  support  the  inventory  of 
tasks.  The  tasks  that  the  matrix  requires  for  its  computations  are  the 
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troubleshooting  and  nontroubleshooting  flightline  tasks  and  intermediate 
shop  repair  tasks.  The  matrix  inputs  include  the  number  of  flightlino  and 
shop  tasks,  average  task  times,  task  frequencies,  and  average  crew  sizes. 
Detailed  operating  instructions  for  the  matrix  may  be  found  in  Reference  20. 

Radar  Test  Synopsis:  The  logistics  engineers  used  the  matrix  to  explore 
the  training  and  technical  manual  coverage  implications  for  the  radar's 
maintenance  tasks.  The  required  data  came  from  the  data  base  that  had  been 
compiled  to  run  the  Reliability  and  Maintainability  Model,  the  Reliability 
and  Maintainability  Cost  Model,  and  the  Page-Estimating  Equations. 

Tables  7  and  8  present  the  output  generated  by  the  matrix  for  the 
baseline  and  improved  radar  configurations,  respectively.  The  matrix 
displays  the  three  basic  types  of  maintenance  tasks  as  column  headings.  The 
left  hand  margin  depicts  the  equipment  subsystems  and  their  indigenous  line 
replaceable  units.  Each  radar  configuration  consisted  of  three  subsystems: 
a  mechanical  one  (coded  AFRME ) ,  a  digital  (coded  AFRDI),  and  a  radio 
frequency  (coded  AFRRF) .  The  ratios  of  numbers  represent  the  emphasis  to  be 
placed  on  training  and  technical  manuals  for  the  maintenance  tasks.  These 
ratios  depict  the  head/book  tradeoffs.  For  example,  the  number  "1"  in  the 
ratio  ’'1/3**  indicates  light  technical  training  emphasis  for  a  set  of 
maintenance  tasks  while  the  "3"  indicates  heavy  concentration  in  technical 
manuals.  The  logistics  engineers  broadly  interpreted  the  matrix  output  in 
Table  7.  It  appeared  that  the  baseline  radar  would  need  more  technical 
manual  support  for  its  flightline  maintenance  tasks  than  for  technical 
training  support.  Shop  tasks  would  require  an  even  division  between 
technical  training  and  technical  manual  emphasis.  Similar  conclusions  were 
drawn  for  the  improved  radar  configuration's  training/manual  trade  offs  in 
Table  8.  It  is  left  as  an  exercise  for  the  reader  to  examine  each 
training/manual  trade  off' for  each  subsystem. 

The  Assessment: 

Criterion  No.  1  -  Can  the  matrix  provide  logistical  information  critical 

to  the  selection  of  one  design  option  over  another? 

It  could  not  be  determined  if  the  information  derived  from  the  matrix 
could  be  of  any  consequence  in  design  option  selection.  The  output  is 
self-explanatory  and  perhaps  ridiculously  simple;  however,  the  manner  in 
which  the  matrix  processes  its  inputs  is  vague.  The  matrix  is  supposedly 
the  mechanism  which  the  Integrated  Task  Analysis  Procedure  uses  for 
coordinating  the  development  of  the  technical  orders  and  training  programs. 

Criterion  No.  ?  -  Are  the  data  elements  well  defined? 

The  data  elements  are  not  explicit.  For  example,  the  matrix  requests 
that  the  user  input  median  performance  standards  and  decimal  factors  for  the 
maintenance  task  inputs  without  further  explanation.  The  user  may  provide 
unsubstatiable  estimates  which  ultimately  undermine  the  matrix  outputs. 


Criterion  No.  3  -  Are  the  theoretical  assumptions  explicitly  stated  and 
do  the  make  intuitive  sense  to  a  logistics  engineer? 


Table  7.  TAM  Baseline  Rader  Assessment 


TRAINING  AIDING  MATRIX 


AFRRF 

AFRRF1 

AFRRF2 

AFRD1 

AFRD11 

AFRD12 

AFRME 

AFRME1 


AFRME2 


Table  8.  TAM  Improved  Rader  Assessment 


TRAINING/AIDING  MATRIX 


FLIGHTLINE 

FLIGHTLINE 

SHOP 

NONTROUBLESHOOT 

TROUBLESHOOT 

REPAIR 

EQUIPMENT 

l 

1 

AFRRF 

1  2  / 

2  / 

H/ 

AFRRF1 

1  /  2 
| 

/  3 

2  / 

/B 

AFRRF2 

1 

/2 

2  / 

1 

1 

/2 

AFROI 

i  1  / 

2  / 

H  / 

AFRDIT 

I  /3 

/  3 

2  / 

/  B 

1 

i 

/  2 

AFRME 

1  3  / 

1  / 

H/ 

AFRME1 

1  /  1 

/  3 

2  / 

/B 

AFRME2 

1 

1 

1 

/  2 

2  / 

1 

/2 
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The  theoretical  statements  are  not  explicit.  The  algorithms  which 
process  the  data  are  not  clearly  evident,  and  therefore,  the  engineers  could 
not  verify  or  dispute  the  statistical  nature  of  the  training  course  versus 
technical  manual  trade  offs.  The  matrix  is  heavily  biased  toward  technical 
order  emphasis. 

Criterion  No.  4  -  Is  the  documentation  complete  and  clear? 

The  documentation  is  not  clearly  presented.  There  are  ample 
opportunities  for  misunderstandings  and  confusion  because  a  data  element 
dictionary  does  not  exist. 

The  Training/Aiding  Matrix  application  ended  the  radar  test 
demonstration  of  ASSET.  The  Training  Requirements  Analysis  Model  and  the 
Personnel  Availability  Model  were  not  used  because  of  numerous  technical 
difficulties  encountered  in  attempts  to  run  them.  The  models  would  have 
been  used  to  evaluate  the  training  and  personnel  requirements  of  the  two 
radar  configurations.  In  doing  so,  the  logistics  engineers  would  have 
evaluated  both  models  against  the  four  criteria.  A  description  of  their 
hypothesized  strengths  and  evident  technical  weaknesses  can  be  found  in 
Reference  25. 

The  logistics  engineers  prepared  a  design  option  decision  tree  to 
summarize  the  supportability  considerations  of  both  radar  configurations. 
Although  the  radar  configuration  assessments  are  incidental  to  the  true 
purpose  of  the  test  and  evaluation  sequence  (i.e.,  the  ASSET  package 
assessment)  the  outcome  may  prove  interesting  from  an  academic  standpoint. 
Figure  8  depicts  the  two  design  options.  Their  reliability  and 
maintainability  figures  of  merit,  life  cycle  costs,  and  training  and 
technical  order  requirements  estimates  are  annotated  on  their  portions  of 
the  decision  tree.  Although  a  decision  tree  can  be  annotated  with  data  down 
to  the  LRU ,  this  tree  depicts  only  the  two  systems  and  their  indigenous 
subsystems,  to  minimize  complexity.  It  appears  that  the  improved  radar 
configuration  does  not  have  a  significant  advantage  over  the  baseline  design. 
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5.0  CONCLUSIONS 


In  summary,  this  test  and  evaluation  of  the  ASSET  technology  involved  a 
number  of  steps.  Pirst,  the  researchers  established  the  hypothesis.  The 
assumption  was  that  ASSET  could  effect  the  consideration  of  logistics 
factors  in  weapon  system  design  during  design  development.  Second,  an 
appropriate  test  bed  was  selected:  a  radar  system.  Third,  evaluation 
criteria  were  devised  to  measure  ASSET'S  operational  readiness  and 
efficacy.  Pourth,  the  ASSET  components  *ere  applied  to  the  test  bed. 
Fifth,  the  components  were  evaluated.  What  remains  is  an  overall  assessment 
of  the  ASSET  package.  Does  ASSET  do  what  it  was  intended  to  do?  What 
procedures  and  tools  within  the  package  can  be  recommended  for  use? 

Criterion  No.  1  (rephrased):  Can  ASSET  effect  the  consideration  of 

logistics  factors  in  weapon  system  design? 

ASSET  can  indirectly  effect  their  consideration.  The  results  of  the 
analytical  tools  may  provoke  inquiry  and  stimulate  further  investigation 
into  the  complex  factors  that  impact  weapon  system  supportability . 

ASSET  does  not  contribute  a  substantial  amount  of  logistical  information 
critical  for  alternative  design  trade  off  decisions  because  of  inherent 
technical  inadequacies  which  undermine  the  methodology  and  ultimately  limit 
its  credibility. 

Criterion  No.  2  Is  ASSET  well  defined? 

This  question  addresses  its  procedures  and  tools.  Only  the  Integrated 
Task  Analysis  Procedure,  Maintenance  Action  Network,  Consolidated  Data  Base 
Development,  and  Design  Option  Decision  Tree  are  well  defined.  The  user  is 
expected  to  have  a  working  knowledge  of  life  cycle  cost  analysis  and 
comparability  analysis.  The  procedures.  Program  Definition  Analysis,  and 
Logistics  Resources  Assessment  resemble  processes  more  than  procedures  and, 
hence,  contain  little  concrete  direction. 

The  models  in  ASSET  that  constitute  its  analytical  tools  range  from 
poorly  defined  to  adequately  defined  in  terms  of  the  data  elements  and  the 
operating  assumptions.  To  be  adequately  defined  means  that  the  models  can 
be  operated  and  results  obtained  even  though  there  may  be  concerns  over  data 
element  definitions  and  operating  assumptions.  Poorly  defined  denotes 
significant  data  element  and  assumption  obscurities.  The  Reliability  and 
Maintainability  Model  and  the  Reliability  and  Maintainability  Cost  Model  are 
adequately  defined  models.  The  Page  Estimating  Equations  program  is 
adequately  defined  but  to  a  lesser  extent  because  of  methodological 
inconsistencies.  Training/Aiding  Matrix  is  poorly  defined  in  data  elements 
and  operating  assumptions.  The  Training  Requirements  Analysis  Model  and  the 
Personnel  Availability  Model  contain  ill  defined  data  elements  and  obscure 
operating  assumptions. 

Criterion  No.  3  Is  ASSET’S  concept  explicit?  Does  it  make  intuitive 

sense  to  a  logistics  engineer? 
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To  answer  the  second  question  first:  Yes,  the  concept  does  make  sense. 
The  major  premise  of  ASSET  is  to  consider  the  human  resources  and  other 
logistics  considerations  in  system  design  as  early  as  possible  to  ensure  the 
development  of  supportable  and  cost-effective  weapon  systems.  Few  would 
dispute  that  premise.  ASSET  conceptualizes  how  it  should  be  applied  to 
achieve  the  desired  effect  yet  it  does  not  describe  the  interrelationships 
that  must  exist  within  the  design  and  logistics  community  to  ensure  that 
logistics  factors  are  seriously  considered  and  that  weapon  systems  are 
developed  which  maximize  supportability . 

Criterion  No.  4  -  Is  ASSET  documentation  complete  and  clear? 

The  documentation  is  a  users  manual  which  has  information  on  the 
procedures  and  instructions  to  operate  the  models.  The  manual  should  be 
supplemented  with  other  technical  literature  to  obtain  a  full  appreciation 
of  the  technology  package.  Supplementary  materials  have  been  identified 
throughout  this  report.  A  potential  ASSET  user  can  apply  the  ASSET 
components  with  minimal  difficulty  using  the  application  manual  and 
supplemental  documentation. 
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6 . 0  RECOMMENDATIONS 


ASSET  does  not  provide  the  complete  system  for  supportability  analysis 
and  conceptual  phase  design  Impact  that  was  originally  envisioned.  However, 
many  of  the  models  have  been  refined  and  used  by  the  Navy  in  their  HARDMAN 
methodology.  The  following  ASSET  models  and  procedures  can  be  usefully 
applied  with  adequate  data  to  appropriate  problems: 

-  Reliability  and  Maintainability  Cost  Model.  If  a  user  were  interested 
in  an  interactive  life  cycle  cost  model  for  fast  feedback  on  recurring 
or  nonrecurring  costs,  it  is  recommended  that  this  model  be  considered 
for  potential  application. 

-  Reliability  and  Maintainability  Model.  This  model  determines  average 
maintenance  times  and  maintenance  manhours  plus  skill  levels  to  be 
associated  with  a  system  design.  Other  models  or  techniques  should  be 
investigated  for  total  resource  requirements. 

-  The  Design  Option  Decision  Tree  and  Integrated  Task  Analysis.  These 
procedures  are  recommended  for  application  in  developing  basic 
maintenance  data  and  structuring  trade  off  considerations. 

-  Page  Estimating  Equations.  Estimating  relationships  such  as  the 

Page  Estimating  Equations  may  be  used  to  develop  technical  order 

baselines  against  which  contractor  technical  order  proposals  may  be 
evaluated.  If  the  per-page  labor  and  material  cost  estimates  are 

known,  then  a  "should-cost"  study  can  be  prepared  to  use  during 

technical  order  procurement  negotiations.  These  equations  were  used 
to  prepare  a  should  cost  study  for  the  F-16  System  Project  Office  in 
1976.  They  were  used  again  to  prepare  a  baseline  for  the  B- 1  Bomber 
technical  order  management  personnel  in  1982.  The  user  must  take  care 
to  reestlmate  the  equations  for  the  specific  application  to  fine  tune 
their  estimating  accuracy  and  precision.  An  axiom  may  be  in  order 
here.  The  power  of  any  analysis  tool  is  derived  from  the  quality  of 
its  data.  Questionable  inputs  will  yield  only  questionable  results. 
Therefore,  it  is  imperative  that  the  potential  user  carefully  review 
the  data  input  requirements  of  each  tool  to  determine,  first,  if  the 
model  can  provide  the  answers  sought  and,  second,  if  the  data  required 
are  available. 

If  ASSET  proved  anything  at  all,  it  demonstrated  the  feasibility  of 
conducting  various  kinds  of  analyses  from  reliability  and  maintainability 
evaluations  to  life  cycle  cost  from  a  single  data  base  that  supports  a 
weapon  system  design. 

A  number  of  factors  that  influenced  ASSET'S  development  altogether 
weakened  its  technical  capabilities  and  may  serve  as  lessons  learned  to 
other  organizations  contemplating  similar  ventures. 
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The  first  factor,  and  perhaps  the  most  critical,  was  that  no  user  was 
identified  at  the  onset  who  had  a  direct  interest  in  the  R&D  effort  and  who 
would  be  bound  to  apply  it  or  any  of  its  components  when  developed. 
Although  potential  users  were  identified  because  of  ASSET*  s  perceived 
relevance  to  their  individual  charters  (e.g..  Air  Training  Command  and 
ASSET'S  training  component)  no  specific  organizations  were  committed  to 
ASSET  and  no  firm  agreements  were  established  for  its  use.  The  second 
factor  is  fallout  from  the  first:  R&D  on  ASSET  was  insulated  from  real  world 
considerations.  Without  close  coordination  with  a  user,  a  technology 
product  may  be  developed  that  will  have  limited  application  in  the  real 
world.  The  Laboratory  developed  ASSET  for  use  as  a  complete  package 
addressing,  in  coordinated  fashion,  the  training,  life  cycle  cost,  personnel 
and  other  logistics  implications  of  weapon  system  development,  not  fully 
realizing  that  the  behavioral  and  political  content  of  weapon  system 
acquisition  did  not  lend  itself  well  to  the  coordinated  consideration  of  all 
those  logistics  factors.  Furthermore,  the  very  detailed  data  requirements 
of  many  ASSET  models  limit  its  applicability  during  the  conceptual  design 
phase  unless  a  determined  effort  is  made  to  uncover  comparable  data.  This 
fact  makes  ASSET  better  suited  for  analyses  during  the  full  scale 
development  phase  of  system  acquisition.  Had  a  user  been  identified  and 
consulted  during  ASSET'S  development,  the  technology  package  may  have  been 
configured  differently  for  realistic  applications. 

The  last  two  factors  are  products  of  hindsight.  ASSET  lacked  and  still 
lacks  a  strong  data  interface  between  its  tools  and  popular  Department  of 
Defense  analytical  models.  The  capability  for  the  transfusion  of  data 
between  a  credible  model  and  a  newly  developed  one  that  could  provide  either 
quicker  responses  or  alternative  analyses  would  have  increased  user 
acceptance.  Lastly,  no  mechanism  was  established  to  update  ASSET.  Over 
time,  computer  programs  need  refinement,  techniques  such  as  regression 
equations  must  be  revalidated  and  procedures  for  problem  solving  must  be 
reexamined  periodically  to  ensure  their  relevance  to  the  issues  that  they 
address.  Currently,  ASSET  components  must  be  closely  examined  for  their 
applicability  to  problems  on  a  case  by-case  basis. 

Since  ASSET  involves  off-line  analysis,  it  may  be  considered  obsolete  in 
view  of  advanced  manufacturing  technologies,  such  as  computer  aided  design 
and  computer  aided  manufacturing  (CAD/CAM) ,  that  will  streamline  product 
design  and  production.  CAD/CAM  allows  the  evaluation  of  numerous  design 
options  without  prototypes.  Additionally,  it  provides  the  design  engineer 
with  rapid  on-line  analysis  capability.  The  ASSET  package  or  a  similar 
package,  may  be  more  effective  if  it  could  be  integrated  into  CAD/CAH 
operations.  It  is  believed  that  logistics  considerations  may  have  a  better 
chance  of  influencing  weapon  system  design  within  this  new  manufacturing 
technology.  Future  AFHRL  R&D  will  investigate  existing  techniques  such  as 
those  in  ASSET  which  can  be  modified  for  access  through  computer  aided 
design  terminals  as  a  more  effective  means  to  impact  system  design.  Such 
techniques  might  provide  more  timely  analysis  of  operational  support 
factors,  such  as  operational  readiness,  life  cycle  cost,  reliability  and 
maintainability,  than  does  the  current  method  of  off  line  analysis  and 
after-the  fact  design  review. 
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1.  Page  14,  Change  last  sentence  at  top  of  page  from:  "Consult  Reference 
11..."  to:  "Consult  Reference  1..." 

2.  Page  14,  Change  last  sentence  In  parentheses  at  bottom  of  page  from: 
"Reference  8..."  to:  “Reference  14..." 

3.  Page  16,  Change  last  sentence  of  first  paragraph  on  page  from:  "In-depth 
descriptions  of  the  generalized  maintenance  action  network  may  be 
obtained  from  Reference  24."  to:  "...from  Reference  9." 

4.  Page  20,  Change  last  sentence  of  first  paragraph  at  top  of  page  from: 

"Consult  References  19,  20  and  26  for  guidance.”  to:  "Consult 

References  19,  23,  and  24  for  guidance." 

5.  Page  20,  Change  last  sentence  In  fifth  paragraph  from  the  top  of  the  page 

from:  "The  user  must  refer  to  other  sources,  such  as  Indigenous 

engineering  personnel  or  References  11  and  19  for  assistance,  to: 
"...or  References  23  and  24  for  assistance." 

6.  Page  22,  Change  last  sentence  In  first  paragraph  at  top  of  page  from: 
"Consult  Reference  21..."  to:  "Consult  Reference  2..." 

7.  Page  22,  Change  last  two  sentences  In  the  third  paragraph  from  the  bottom 

from:  "Detailed  Information  regarding  the  Reliability  and 

Maintainability  Model  may  be  obtained  from  Reference  20."  to: 

"...Reference  10." 

"Additional  sources  for  the  Reliability  and  Maintainability  Model  are 
References  22  and  23."  to:  "An  additional  source  for  the  Reliability 
and  Maintainability  Model  Is  Reference  11." 

8.  Page  23,  Change  last  sentence  In  fifth  paragraph  from  top  of  page  from: 
"Detailed  descriptions  of  these  reports  may  be  obtained  from  Reference 
20."  to:  "Detailed  descriptions  of  these  reports  may  be  obtained  from 
Reference  10." 

9.  Page  23,  Change  last  sentence  In  third  paragraph  from  bottom  from:  "A 
print-out  from  the  data  Inputs  and  data  values  may  be  obtained  from 
Reference  20."  to:  "...from  Reference  1." 
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